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(54) Method for decorating a virtual model 

(57) A method for decorating a virtual world model 
first builds a physical model from a plurality of building 
blocks. Each building block includes a microcontroller 
coupled to a plurality of connectors. The connectros are 
for physically and electronically connecting the blocks in 
a three-dimensional structure to form the model. An 
arrangement of the blocks in the model is derived by 



connecting the model to a host computer. The anange- 
ment is expressed as a set of logical axioms. The set of 
logical axioms is processed by a logic program to iden- 
tify large scale structural elements of the model, and 
decorative attributes are assigned to the large scale 
structural elements. 
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Description 

REU) OF THE INVENTION 

5 [0001] This invention relates generaify to virtual reality models, and more particularly to decorating virtual models 
derived from physical models. 

BACKGROUND OF THE INVENTION 

10 [0002] In U.S. Patent 4,275,449 "Modeling Arrangements," Aish describes a set of building blocks as a computer 
input device for architectural applications. The blocics were geometric solids with connectors on some of the faces, and 
could be changeably interconnected to form different modeling arfangennents whose geometric structure could be read 
by a computer. Each block had an identifier, which when used as an index into a file of information at>out the blocics, 
permitted 3-D renderings to be made of the physical model. Aish devised an approach to reading out the structure of a 

75 modeling arrangement that kept the circuitry in each block to a minimum. A host computer directed a search of the 
structure, selecting one block at a time. That block's identity was read, then neighbors detected and control passed from 
that block to a neighbor, and so on, until an exhaustive search of the structure had been completed. 
[0003] Evans, in "Intelligent Building Blocks," Architect's Journal, 30 January 1 985, pp. 47-54. mentions that other 
infomnation, such as material properties and costs, could also be associated with such blocks, permitting the computer 

20 to prepare various architectural analyses and reports about the modeled structures. 

[0004] Frazer. in "An Evolutionary Architecture," Architectural Association, 1 995, describes a more ambitious series 
of prototypes of machine-readable modeling tools. In general their approach to reading the modeling structure followed 
Aish's, although they tried several different kinds of building elements, and used them for a variety of applications. In 
one embodiment, each of Fra2er*s blocks had eight bits of state reflected in eight LEDs that could be controlled by a 

25 host computer. One of the blocks was equipped with six mercury tilt-switches to determine the orientation of the entire 
model. Another block had magnet-sensitive reed-switches embedded in external cladding panels. As the computer 
came to poll that block for its identify, the state of these switches could affect the result in a way that would in turn affect 
the virtual model's rendered appearance. 

[0005] Frazer also developed a modeling kit whose elements corresponded to the components used in kits for 
30 building actual modular homes. The miniature modeling kit included a variety of elements such as wall panels, doors 
and windows. Software on the computer drew plans, gave feedback on planning errors, estimated costs and energy 
consumption, etc. 

[0006] Dewey et al., in "Geometry-Defining Processors for Partial Differential Equations," B.J. Alder (ed.) "Special 
Purpose Computers," Academic Press, 1988, pp. 67-96, describe a set of 3-D blocks similar to some of those built by 

35 Frazer's group, but with a different application in mind. The motivation tor their geometry-defining processors was to 
build a re-configurable parallel computer for finite-element simulations of systems studied in mechanical engineering. 
Thus, the connection geometry of the parallel processing elements could match the geometry of the underlying physical 
system being modeled, and thereby use the available communications bandwidth more efficiently. Because the princi- 
pal goal was engineering computation, each building element contained a commercially available microprocessor. 

40 [0007^ Other related work is described by Gorbet et al. in "Triangles: A Physical-Digital Construction Kit," Proceed- 
ings of Designing interactive Systems: Processes, Practices Methods and Techniques, August 1997, pp. 125-^28, and 
in "Triangles: Tangible Interface for Manipulation arid ExplorBtidn of Digital Information Topography," Proceedings of 
CHI 98, Apr 1998, pp. 49- 6. In the "Triangles" system, the-basic building elements are triangles. Each triangle is a pla- 
nar, plastic equilateral triangle with an embedded rhicroprocessor. The triangles connect to each other physically and 

45 digitally with magnetic, electrically conducting connectors. When connected to each other, the triangles can be tiled on 
a flat surface, or folded over into more complex surface topologies. When the triangles are connected, information about 
their identities is exchanged, and messages can be relayed lo a :host computer. In this way, an application running on 
the host can determine relationships between the connected pieces, and specific connections can trigger specific dig- 
ital events. Typical applications include visuall progranirh)ng,:Visua^ and pattern formation. 

so [0008] Key attributes desired of self -describing construction kits include: scalability - the ability to build large struc- 
tures containing hundreds of building elements, configurability - the ability to connect building elements in rich and var- 
ied ways, interactivity - the ability to interact physically and electrically with a constructed artifact, and presentation - the 
ability to design customized and stylized visual and physical interpretations of constructed artifacts. Known prior art 
building block systems lack integrated solutions that address these key attributes. 

55 

SUMMARY OF THE INVENTION 

[0009] Only skilled people know how to use graphics modeling packages, such as a CAD/CAM system, but every- 
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one can build things out of blocks. Starting from this premise, and with the goal of developing accessible modeling tools 
for building and populating virtual worlds, the invention provides a novel object-modeling system. The system includes 
building blocks that setf-describe the geometric structures into which they are assembled. Each building block contains 
a microcontroller, and can communicate with the other blocks to which it is physically connected. 

5 [0010] The invention provides a novel architecture for a distributed computer system comprising self describing 
building blocks with embedded microprocessors (microcontrollers). Each self describing building block is formed of an 
enclosure having a top surface and a bottom surface. An array of m by n radially symmetric connectors are arranged 
on the top surface and on the bottom surface of the enclosure, wherein both m and n are greater than one if a rigid 
structure is required. The connectors are configured to carry power and data signals. A microcontroller, including a 

10 memory, is mounted in the enclosure. The microcontroller is coupled to each of the connectors. The microcontroller 
includes communication means for exchanging data messages using any of the connectors. 

[0011] The connectors enable a plurality of the blocks to be arranged in a rigid three-dimensional structure. This 
structure can be recovered by a distributed computation perfonmed by the blocks, and passed to a host computer. The 
host computer can make the structure available for various applications, including virtual-reality computer games, infor- 

15 mation management for buildings, and artistic expression. 

[0012] From the collected block connectivity data, and presorted or editable block attributes, the host can recover 
the geometric structure of the assembled blocks. The structure can then be rendered in various styles, ranging from a 
literal rendition, to decorative interpretations in which structural elements are identified automatically and augmented 
appropriately After being rendered, the virtual models are available for viewing and manipulation by the user. The auto- 

20 matically decorated models can also be "replicated" using 3D stereolithography. 

[001 3] After the block connectivity data has been collected, each block can communicate with the host, and with 
the other blocks. Sensors in the blocks can report their status, and transducers such as lights, speakers, motors, etc., 
can be controlled. For example, the blocks can be assembled into a model of an actual building, with sensors in the 
model being used to control the lighting in the building, and sensors in the building being used to control the conre- 

25 spending lights in the model. 

[0014] The geometric arrangement of the blocks, as well as sensor data, and transducer controls can also be 
shared over a network, e.g.. the Internet, permitting collaborative design, remote monitoring, and multi-user game play- 
ing, for example. 

[0015] In contrast to the prior art, our system concurrently achieves scalability, configurability, and interactivity, as 
30 well, as a unique capability ;to enhance constructed artifacts through automatic, intelligsnt decoration. This is accom- 
plished using a physical form factor that allows a rich and varied connectivity of building elements; a microprocessor- 
based, distributed, packet-switching architecture that facilitates efficient and robust computation of connectivity, and the 
autonomous operation of biiilding elements during interactive use; and the automatic interpretation of constructed arti- 
facts fpr the purpose of visual and physical decoration. 

35 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0016] 

40 Figure 1 is a top perspective view of a construction element or building block according to the invention; 

Figure 2 is a bottom perspective view of the block of Figure 1 ; 
Figure 3 is a side view of a circuit board mounted in the block of Figures 1 and 2. 

45 

Figure 4a is a flow diagram of the process used by the building blocks for determining the connectivity of an 
arrangement of blocks; 

Figure 4b is a block diagram of a message routed among the blocks; 

50 

Figure 5 is a perspective view of an arbitrary rigid arrangement of building blocks according to the invention; 
Figure 6 shows a literal graphic rendition of the arrangement of Figure 5; 
55 Figures 7-10 are alternative graphic renditions of the arrangement of Figure 6; 

Figure 1 1 illustrates a checkerboard signaling pattem; 
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Figure 12 is a flow diagram of a process for determining the connectivity of an arrangement of blocks: 

Figure 13 is a diagram illustrating the interaction among a model world made from the construction elements, a 
graphic virtual world of the model world, and a physical work represented in the model and virtual worlds; 

5 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
Block Structure 

10 [0017] Figures 1-2 shows a construction element 100 according to our invention. Figure 1 is a top view, and figure 
2 IS a bottom view. The element 1 00 is in the form of a 2x4x2 building block. Here, 2x4x2 refers to the arrangement of 
the sixteen connectors on the top and bottom of each block. It should be noted that in general the invention admits any 
m X n array of connectors on the top and bottom of the t)lock. However, to forrh a rigid structure both m and n need to 
be greater than one. 

75 [0018] Each block has a unique block identification number (BID), and each connector has an associated connector 
identification number (CINi, .... CIN-ie)- 

[0019] The block 100 is a two-part plastic enclosure having a top part 101 and removable bottom part 102. The 
parts can be fixed together with screws. Holes are formed in the top and bottom to mount connectors 1 03-1 04. 
[0020] The eight top holes are tor plugs 1 03 (CIN-i, .... ClNs), and the eight bottom holes mount jacks 104 {CIN9, 
20 CIN-ie). To allow the blocks to be connected together in a wide range of orientations, we make the connectors radially 
symmetric, for example, round. 

[0021] The dimensions of the blocks (mxn), and the locations of the radially symmetric plugs and jacks are such 
that multiple blocks can be arranged into the same kind of 3D structures that one could create with LEGO™ building 
blocks. 

25 [0022] Each of the connectors 103-104 has two conductors (lines). However, instead of using one conductor for 
power and one for ground, which is the normal usage for a DG power connector, we use an inner conductor as a signal 
line 105 for bidirectional data communications, and an outer conductor or sleeve 106 for power distribution. In an alter- 
native embodiment described below, a single conductor is used for both power and data signal. In this case, the power 
signal is modulated to carry the signal. 

30 [0023] The outer conductors 1 06 are wired so that adjacent connectors- alternate power and ground signals on their 
outer sleeves, in a pattern similar to one described in U.S. Patent 4,883,440 "Electrified Toy Building Block with Zig-Zag 
Current Carrying Structure" issued to Belli. Thus, every block will have at least one connection to power and one to 
ground in any typical structure built with the blocks. 

[0024] When the blocks are connected, there is no way to tell a priori which sleeves are connected to power and 
35 which sleeves are connected to ground. We solve this problem as shown in Figure 1 1 . All of the sleeves for one of the 
polarities 1101 are connected to one of the inputs of a full-wave bridge rectifier 320, see Figure 3, and all of the sleeves 
for the other polarity 1 102 to the other input of the rectifier 320. The output of the bridge rectifier is a supply voltage with 
a known power and ground polarity. Note that there is a voltage drop as the power passes through the full-wave bridge 
rectifier 320 and as power passes through the connectors 1 03 and 1 04. 
^ [0025] This su pply voltage is then fed into a voltage regulator 330 that outputs con-ectly reg ulated power fo r the inte- 
grated circuits used within the block's circuitry. The voltage regulator ensures that the correct voltage is supplied to the 
circuitry independent of the input voltage to the regulator, so long as voltage is sufficiently high. This allows higher volt- 
age, unregulated power to be distributed from block to block. This in turn allows a greater distance from the power 
source to the power consumer, and thereby enabling the construction of larger structures. 
45 [0026] Data communication according to our invention, described in greater detail below, is done without the use of 
a global communication bus. Instead, we use a message passing protocol via the signal lines 105. 
[0027] Two holes 107 are formed in each side of the block. A light emitting diode (LED) is mounted behind each 
hole so that there is a visual indication of the block's operation. 

[0028] As shown in a side view in Rgure 3, a circuit board 300 is mounted in the block 100. The connectors 103- 
50 1 04 are fixed to the circuit board, as are the LEDs 301 . The connectors form the physical mounting baste for the circuit 
board. A microcontroller 310, the rectifier 320, and the voltage regulator 330 are mounted on the board. We use a 
Microchip™ Technology Inc. PIC16C77 microcontroller. The microcontroller includes a random access memory. The 
board 300 also includes pads tor the connector plugs and jacks, and assorted analog elements, i.e., resistors, capaci- 
tors, and a crystal to generate the clock for the nriicrocontroller. 
55 [0029] Several pads in the periphery of the circuit board are left open to accommodate additional transducers and 
sensors inside the block, such as a speaker 1 10 shown in Figure 2, motors, RF, IR, ultrasound, and microwave trans- 
mitters, LED, LCD, CRT, and other display devices, cameras, microphones, proximity, presence, motion, and touch sen- 
sors 120, RF, IR, ultrasound, and microwave receivers, and the like. Alternatively, the current board can be trimmed to 
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fit inside a 2x2x2 building block without affecting its basic functionality. 

[0030] The microcontroller 310 has thirty-three I/O pins of which sixteen are used for data communication via the 
signal lines of the connectors. The microcontroller includes an eight-bit RISC CPU, 8K of 1 4-bit words of read-only pro- 
gram memory (PROM). 368 8-bit bytes of data memory (RAM), a hardware Universal Synchronous/Asynchronous 
5 Receiver/Transmitter (USART), and interrupt handling. The data memory is used to store temporary data and connec- 
tivity information during operation. 

[0031] We program the controller in assembly language to achieve execution efficiency^ and to save code space. At 
startup, the program and data in each block's microcontroller are identical other than the block's unique identification 
number (BID). 

10 

Operation of Assembled Structure 

[0032] The operation of our invention is described with reference to Figure 4a. An arbitrary number of blocks are 
connected to each other to fomn some three-dimensional structure 400. Because of the constructive versatility of the 
15 blocks having an mxn an^y of radially symmetric connectors, a pair of 2x4x2 blocks can be connected in 1 84 different 
configurations, and one 2x4x2 block can be connected to as many as twelve others. 

[0033] The geometry of the three-dimensional structure is determined using a message-routing protocol. The low- 
level details of the protocol are described below. At a high level, the protocol operates in three phases. Phase 1 deter- 
mines connectivity, Phase 2 determines plug-to-jack con-espondences, and in Phase 3, the connectivity database is 
20 sent, or "drained," to, for example, a host computer 480. The procedures that comprise the three phases operate in par- 
allel 499 in the various blocks. The host computer can be a PC, workstation, or the like. 

[0034] After these three phases have completed, the bricks enter a final phase known as Phase R ("R" for Routing). 
This phase allows the blocks to communicate with each other or with the host, and for the host to communicate with the 
blocks. The blocks function as general-purpose network routers and are also able to autonomously determine the state 
25 of sensors and alter the state of transducers during this phase. 

Phase 1 

[0035] Operation commences when we supply a power and ground signal on lines 401-402 to a drain block 403, 
30 described below. From the drain block(s), power is distributed to all of the blocks. When the blocks are powered on, they 
immediately enter Phase 1 (41 0). During Phase 1 , the blocks determine their connectivity 41 1 in parallel. Lacking a glo- 
bal communication bus, the switching on of power is the only source of synchronization, which is necessarily approxi- 
mate because of small delays in propagating power throughout the entire structure. 

[0036] All. 1 6 signal lines of each block, i.e., 8 connectors 1 03 on top and 8 connectors 1 04 on the bottom, are nor- 
35 mally held high by pull-up resistors. Phase 1 is initiated by having each block pull its top signal lines (those in the plugs) 
low in response to the power signal. 

[0037] Each block then tests its bottom signal lines (the lines 105 in the jacks 104) to determine and store which 
have been pulled low by some other connected block. After a short delay, to ensure that the approximately synchronized 
blocks do not try to drive shared signal lines simultaneously in both directions, this test is repeated with the roles of the 
40 top and bottom signal lines reversed. 

[0038] Thus when Phase 1 completes, each block has identified in parallel which of its signal lines are connected, 
i.e., which connectors are attached to other blocks, but not to which specific other blocks or connectors. 

Phase 2 

45 

[0039] After another short delay, each block enters Phase 2 (420). This phase has a first bottom-to-top halt and a 
second top-to-bottom half. During this phase, blocks communicate with neighboring blocks over the connected signal 
lines identified in Phase 1 to determine plug-to-jack correspondence information 421. At the start of the first half of 
Phase 2, each block first listens on its connected bottom signal lines for transmitted messages that contain the BID of 
50 the transmitting block, BIDj, and the connector number (CIN) of the connector over which that block is transmitting, 

[0040] Note, transmitted messages may be missed when the receiving block is busy when transmission com- 
mences. Electrical interference, or noise, may also corrupt a message. Therefore, all messages are transmitted using 
a check-summed, acknowledged protocol, and unacknowledged transmissions are retried a predetermined number of 
55 times after appropriate timeouts. 

[0041] The receiving block stores the message content along with its own BID number, BIDr, and the connector 
number over which it received the transmission, CINr, in its memory, e.g., (BIDj, CINy, BIDr, CINr). This datasetforms 
a complete record of a single connection between two blocks. A database 450 of these records stores the local connec- 
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tivity information in each block. 

[0042] After a block has successfully received transmissions on all of its connected bottom lines, it begins transmit- 
ting on each of its connected top lines, iterating in order of its connector numbers. Connectivity information therefore 
flows initially through the block structure from bottom to top. with the potential for significant parallel communication. At 
5 the completion of this parallel "procedure, ieach block knows to which conhectdr of which block each of'lts own bottom 
connectors is attached. 

[0043] The procedure is repeated during the second halt but with blocks listening on their top connected fines and 
transmitting on their bottom connected lines. Thus at the end of Phase 2. each block has acquired and stored in its 
memory complete knowledge about all of its connected lines: the BIDs of the connected pair of blocks, and the corre- 
10 spending connector numbers by which they are attached. 

[0044] Each connected line that is processed successfully in Phase 2 is designated as vaiid; unconnected lines are 
deemed invalid. 

Phase 3 

15 

[0045] In Phase 3 (430), the connectivity information (BIDs and ClNs) is communicated to a host computer via the 
drain block 403. which includes a serial communication connection to the host computer. In addition to mediating com- 
munication between a block structure and the host, the drain also supplies power to the connected blocks, and may be 
attached to any part of a block structure. In other words, the drain can be configured as block 100 with a serial connec- 
20 tion. 

[0046] During Phase 3, all blocks listen on all of their valid lines for messages. When a request-to-drain message 
is received by a block on a particular connector, that connector is designated a "drain" connector. The request-tcxlrain 
message can be generated by the drain block 403 and first sent to some block connected to it, or the requeshto-drain 
message can be generated tjy the host computer and first sent to the drain block 403. Beginning with Phase 3, a drain 
25 block functions by reliably forwarding messages to a block connected to it; its existence is transparent to the algorithm. 
In response to receiving the request-to-drain message, a block transmits connectivity messages containing all stored 
plug-to-jack correspondence information 421 on its "drain" connector, the one from which it received the request-to- 
drain message. 

[0047] After the block has successfully completed transmitting the connectivity messages, the block forwards the 
30 request-to-drain message on a vaiid line with the lowest connector number, and the block enters a message-forwarding 
mode. 

[0048] If the block receives a connectivity message in response to forwarding the mquest-to-drain message, then 
the block forwards that message on its drain connector If the block receives any subsequent /Bquesf-fo-dra/n mes- 
sages, then the block responds with a done message. If the block receives the done message, then the block forwards 
35 the request-to-drain message on a valid line with a higher connector number. If all valid lines have been processed, 
then the block sends a done message on its drain connector. 

[0049] As stated above, the first request-to-drain message can be injected into the structure 400 by the drain block 
.403. The requests then percolate through the block structure in a preorder, depth-first traversal. Although this traversal 
is performed sequentially, that is, with only one block draining at any point in time, the forwarding of messages towards 

40 the drain is pipelined, thereby achieving parallelism in this phase as well. 

[0050] At the end of Phase sTthe host 480 has compTete^onnectivityTnformatio for the block structure. In fact, the 
host has redundant connectivity information, because each connection is reported twice, once by each of the two con- 
nected blocks. This redundancy contributes to the robustness of the system, but it can be eliminated for efficiency by 
eliminating the second half of Phase 2. Because block structures are rigid, their geometry can be inferred deterministi- 

45 cally from complete connectivity data. Using geometrical data so inferred, various renderings of block structures can be 
computed, as described below. 

Phase R 

50 [0051] Phase R (440) Implements a scalable and responsive approach to providing Interactivfty. Blocks autono- 
mously report events, rather than being polled for state changes. Using the route-to-drain found in Phase 3, messages 
about events are communicated in a store-and-forward fashion through a chain of blocks to the drain block, and on to 
the host. 

[0052] In Phase R 440, blocks listen for general-purpose messages. Each message is check-summed, and suc- 
55 cessfully received messages are acknowledged. After a timeout, an unacknowledged message is retransmitted until 
successfully acknowledged. 

[0053] As shown in Figure 4b, messages 460 are composed of a sequence of packets 461 . Each packet contains 
a prefix 462 which includes a packet sequence number 466 and an optional route-to-drain indication 467. The packet- 
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sequence number is used to guarantee that packets are received successfully and in the correct order If the route-to- 
drain indication is present, then the packet is fonwarded to the receiving block's drain connector without further process- 
ing. Each packet also includes a header 463. content 464, and a checksum 465. 

[0054] Phase-R packets are identified by packet type 468 in the header. The header also stores the length 469 of 
5 the packet. The accepted packet types include: (1) a ReadAA/rite- RAM/registers packet. (2) a RAM/registers-contents 
packet, (3) an Alter-LEDs packet, (4) a Play>music packet, (5) a Route-packet-following-specified-path packet. This last 
packet type allows other packets to be sent to any other block. 

[0055] The specified path is detennined by listing a sequence of Connector Identification Numbers (CIN) over 
which the packet should be fonwarded. Using a form of "worm-hole" routing, each block sets up a forwarding path to the 

10 first CIN specified in the list and then removes that CIN from the list that is fonwarded to the next block. 

[0056] Phase R monitors the state of attached sensors, and upon determining a change, autonomously sends a 
RAM/registers-contents packet to the host, using the route-to-drain indicator, to notify the host of the change. Thus , the 
host is made aware of the state of all sensors on all of the blocks all of the time. The host may send packets to specified 
blocks, using the Route-packet type, to cause the blocks to change the state of attached transducers. Thus, the host is 

15 able to control accessories attached to the blocks. 

[0057] As shown in Figure 4b, a message 460 is composed of one or more packets 461 . Packets are composed of 
a sequence of separately synchronized bytes. Each packet contains an optional prefix byte 462. a message header 
463, message-contents bytes 464, and a checksum 465. The prefix stores a sequence number 466 and a route-to-drain 
indication. The header stores a packet type 468 and a packet length 469. The content 464 can be routing information 

20 (CINs), or actual message-specific data. 

[0058] Packets are acknowledged, and unacknowledged packets are retransmitted using an adaptive timeout algo- 
rithm. Packets are transmitted in a pipelined fashion. Thus, the system using the protocol, as described herein, emu- 
lates a self-configuring, store-and-f onward computer network. This form of architecture facilitates transducers that 
require active control, such as speakers or motors, and sensors that require data buffering, such as microphones and 

25 cameras. 

Graphic Rendering of Structure 

[0059] As shown in Figure 4a. the host 480 has access to a database (DB) 490. The database stores the BID 491 
3o and attributes 492 for each biock in the structure 400. The attributes can include shape, size, color, texture, and other 
physical or graphic information associated with the identified block. In other words, the attributes 492 can give the 
blocks additional characteristics. The attributes can be assigned when the blocks are given their unique BIDs. The data- 
base can be edited. This process is illustrated in Rgure 12. 

[0060] In Figure 12, a database 1200 stores the brick specific information. Connectivity information 1210 is 
35 received from Phase 3 as described above. Step 1220 infers the placement of the individual blocks. At this point, files 
1231-1233, in various standard formats, describing the structure 400 can fc>e generated. 

[0061] A graphics-rendering application 481 executing on the host can now graphically render 1240 the 3-D geom- 
etry 500 of the structure on an output device 485, and a user can perform Phase R interactions 1250 via a user inter- 
face. The application 481 can be any common 3D modeling system. For example, Figure 6 shows a graphb 600 
40 generated by the host from the physical block structure 500 shown in Figure 5. Note, that the rendered shape of the 
blocks can be different from the physical shape of blocks 100. For example in Figure 6, the blocks are made to look Rke 
lego'*' blocks. 

[0062] In Figure 10 the same structure 500 has been incorporated as a building 1000 in the well-known Quake"^ 
game. In this rendition, it actually becomes possible to "navigate" through the structure in a virtual-reality environment. 
45 In this case, the attributes given to the blocks include game effects, such as explosive walls, trap doors, electrified walls, 
etc. 

Decorative Rendering 

50 [0063] In addition, we generate a description of the block structure as a set of logical axioms 493 stored in the data- 
base 490 of Figure 4a, one axiom to assert the location, orientation, and identification (494) of each block. These axi- 
oms can serve as input to a logic-program application written in, for example, the Prolog language, to identify larger 
scale structural elements of a block structure through unification -driven pattern matching, e.g., the walls and roof of the 
structure 500 when the structure is to be interpreted as a building, see Appendix A and B for detailed examples. 

55 [0064] Other structural elements identified include windows, doors, corners, roof apexes, etc. Recognized struc- 
tural elements can then be colored and textured distinctively, and decorated with additional ornamental geometry, to 
enhance the visual appearance of the rendered model. 

[0065] In Figure 7, the graphic 700 shows the identical physical structure rendered as a thatched roof cottage: note 
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the brick texture, roof blocks modified to slope smoothly, and rendered with a mottled texture, and the additional fascia 
around the base of the roof. 

[0066] In Figure 8, the same blocks are rendered as a castle 800. The crenellated walls are built out of randomly- 
shaped, colored blocks of stone to create a hand-made appearance. Bars have been placed in the windows. A flagpole 
has been placed at the apex of the roof, and red turrets have been added at the outside corners. 
[0067] In Figure 9, the same blocks 400 or 500 become a wooden fort 900. 

[0068] The pattern -matching rules can be expanded to recognize more structural elements of buildings, and to sup- 
port additional rendering styles. In addition, interpretive and decorative rules for other modeled artifacts, such as vehi- 
cles or organic forms, can be devised. 

[0069] Once rendered in a decorative fashion, the augmented geometry of a structure can be transformed into an 
appropriate format for 3-D printing, e.g., via stereo lithography Thus multiple decorative versions of a block structure 
can be realized physically. 

Multi-World Interactivity 

[0070] The blocks 100 can be equipped with proximity and touch sensors so that the user can interact with the 
blocks in real-time. Sensor data is conveyed to the host computer via the Phase-R protocol described above. Transduc- 
ers in the blocks are also controlled via Phase-R messages. 

[0071] As shown in Figure 13, these features make possible several monitoring and control applications that relate 
the real physical world 1301 , a model world 1302, and a virtual worid 1 303 connected by a networi< 1 304. For example, 
a block-structure model can be created for a real-world building. Virtual representations of the building can be viewed 
on a computer display. By manipulating switches or sensors on the block structure, aspects of the real-wortd building 
such as lights and thermostats can be controlled. The status of real-worid sensors can be reflected in the block model, 
e.g., by turning on LEDs or activating the speaker. In both instances, state and behavior data can also be depicted in 
the virtual model. More generally, any change of state in any one of the worids can be reflected in any other worids. 

Summary 

[0072] In summary, our invention provides a tangible interface that allows users to fashion and manipulate virtual 
object models by simply building a physical structure. Object recognition allows a host computer to intelligently manip- 
ulate and augment the structures built by the user. 

[0073] It is to be understood that various other adaptations and modifications may be made within the spirit and 
scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications 
as come within the true spirit and scope of the invention. 
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APPENDIX A 



% wall/1 finds sets of blocks that form walls. A wall is defined to be 
% a contiguous set of blocks that lie flush against some vertical plane, 
% and that comprise a given fraction of the structure. 

wall(WALL _B LOCKS) :- 
structure _bbox(X _MIN, X _MAX, _, _, Z _MIN, Z _MAX), 
candidate _planes(X _MIN, X _MAX, Z _MIN, Z _MAX, U, V, W, R). 
lies _flush _against(U, V, W, R, PLANE _BLOCKS), 

contiguous _subsets(PLANE _BLOCKS, PLANE _BLOCKS _SUBSETS), 
memberOVALL _BLOCKS, PLANE _BLOCKS _SUBSETS), 
big _enough(WALL _BLOCKS). 

% wall _tops/l finds the blocks that are the tops of walls, 
wall _tops(WALL _TOPS) :- 
findali ( BLOCK, 

( wall(WALL_B LOCKS), 
member(BLOCK, WALL_BLOCKS), 
not _overhung(BLOCK, WALL _BLOCKS)), 
WALL_TOPS). 
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APPENDIX B 

% roof _blocks/l computes the set of blocks comprising the roof, which is 
% defined to be those blocks that rest directly or indirectly on the tops of 
% walls. The indirectly resting blocks are computed by grow _TOoffl. 

roof_blocks(ROOF_BLOCKS) :- 
findall( =BLOCKl, 
( =wall _tops(WT _BLOCK.S), 
member(WT _BLOCK, WT _BLOCKS), 
on _top _of(BLOCKl, WT _BLOCK)), 
BASE _BLOCKS _BAG), 
setof( =BLOCK2, 
member(BLOCK2, BASE ^BLOCKS _BAG), 
BASE _BLOCKS), 
grow roof(BASE _BLX>CKS, ROOF _BLOCKS). 

grow _roof(NASCENT _ROOF _BLOCKS, FINAL _ROOF _BLOCKS) :- 
member(BLOCKl, NASCENT _ROOF_B LOCKS), 
on _top _of(BLOCK2, BLOCKl), 

not member(BLOCK2, NASCENT _ROOF _BLOCKS), 

grow _roof([BLOCK2 1 NASCENT _ROOF_BLOCKS], FINAL _ROOF 

BLOCKS), 
t . 

grow_roof(ROOF_BLOCKS, ROOF_BLOCKS). 



Claims 

1. A method for decorating a virtual world model, comprising the steps; 

building a physical model from a plurality of building blocks wherein each building block includes a microcon- 
troller coupled to a plural'ity of connectors, the connectors for physically and electronically connecting the 
iAocMs in a thee-dimensional structure to form the model; 

deriving an arrangement of the blocks in the model by connecting the model to a host computer; 
expressing the arrangement as a set of logical axioms; 

processing the set of logical axioms by a logic program to identify large scale structural elements of the model; 
and 

assigning decorative attributes to the large scale structural elements. 

2. The method of claim 1 further comprising rendering the attributed arrangement on an output device. 
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for physically and electronically connecting the blocks 
in a three-dimensional structure to fonn the model. An 
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connecting the mode! to a host computer. The arrange- 
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logical axioms is processed by a logic program to iden- 
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